BUG: Construct Timestamp with tz correctly near DST border #21407

mroeschke · 2018-06-10T07:45:52Z

closes #19970
closes #20854

tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

So I think the reason we need to convert a Timestamp to datetime branch is because of how Timestamp behave with normalize explained by this #18618 (comment)

pep8speaks · 2018-06-10T07:45:55Z

Hello @mroeschke! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on June 13, 2018 at 03:17 Hours UTC

codecov · 2018-06-10T16:51:48Z

Codecov Report

Merging #21407 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #21407   +/-   ##
=======================================
  Coverage   91.89%   91.89%           
=======================================
  Files         153      153           
  Lines       49600    49600           
=======================================
  Hits        45580    45580           
  Misses       4020     4020

Flag	Coverage Δ
#multiple	`90.29% <ø> (ø)`	⬆️
#single	`41.86% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab668b0...ae9dc7b. Read the comment docs.

jbrockmendel · 2018-06-10T22:12:06Z

pandas/_libs/tslibs/conversion.pyx

        if ts.tzinfo is not None:
            if hasattr(tz, 'normalize') and hasattr(ts.tzinfo, '_utcoffset'):
+                # tz.localize does not correctly localize Timestamps near DST
+                # but correctly localizes datetimes
+                if hasattr(ts, 'to_pydatetime'):


Elsewhere we've used not PyDatetimeCheckExact(ts) as a standin for isinstance(ts, pd.Timestamp). This might also be use case for the nanos kwarg.

Thanks @jbrockmendel. Seems clearer than trying to get Timestamp to quack

jbrockmendel · 2018-06-11T16:15:22Z

pandas/_libs/tslibs/conversion.pyx

        if ts.tzinfo is not None:
            if hasattr(tz, 'normalize') and hasattr(ts.tzinfo, '_utcoffset'):
+                # tz.localize does not correctly localize Timestamps near DST
+                # but correctly localizes datetimes


IIRC tz.localize (or possibly tz.normalize if thats relevant) operates by calling ts methods. Is this potentially something we can fix "higher up" in Timestamp?

mroeschke · 2018-06-12T05:55:17Z

pandas/_libs/tslibs/conversion.pyx

@@ -347,10 +347,9 @@ cdef _TSObject convert_datetime_to_tsobject(datetime ts, object tz,
    if tz is not None:
        tz = maybe_get_tz(tz)

-        # sort of a temporary hack
        if ts.tzinfo is not None:
            if hasattr(tz, 'normalize') and hasattr(ts.tzinfo, '_utcoffset'):


This check might be superfluous since datetimes natively have the astimezone method. Then this whole block might be able to be simplified.

hmm, you can try that and see if it works. maybe add a comment anyhow

jreback · 2018-06-12T11:17:00Z

doc/source/whatsnew/v0.23.1.txt

@@ -129,6 +129,10 @@ Bug Fixes
 - Bug in :func:`concat` where error was raised in concatenating :class:`Series` with numpy scalar and tuple names (:issue:`21015`)
 - Bug in :func:`concat` warning message providing the wrong guidance for future behavior (:issue:`21101`)

+**Timezones**


move to 0.23.2

jreback · 2018-06-12T11:17:46Z

pandas/tests/frame/test_operators.py

@@ -992,6 +992,16 @@ def test_boolean_comparison(self):
        result = df == tup
        assert_frame_equal(result, expected)

+    @pytest.mark.parametrize('tz', [None, 'America/New_York'])
+    def test_boolean_compare_transpose_tzindex_with_dst(self, tz):


can you move to test_timezones

jreback · 2018-06-12T11:20:28Z

pandas/_libs/tslibs/conversion.pyx

@@ -347,10 +347,9 @@ cdef _TSObject convert_datetime_to_tsobject(datetime ts, object tz,
    if tz is not None:
        tz = maybe_get_tz(tz)

-        # sort of a temporary hack
        if ts.tzinfo is not None:
            if hasattr(tz, 'normalize') and hasattr(ts.tzinfo, '_utcoffset'):


hmm, you can try that and see if it works. maybe add a comment anyhow

jreback · 2018-06-13T10:52:37Z

thanks @mroeschke nice clean up!

can you review any outstanding issues that involve dst. this may go a long way towards solving them (if so, then pls PR for tests).

…v#21407)

(cherry picked from commit bc4ccd7)

…v#21407)

mroeschke added 3 commits June 9, 2018 14:48

BUG: Correctly localize Timestamp near DST

c5566d5

fix pandas-dev#19970

5d207ea

Add whatsnew and more comments

f4eb827

mroeschke added the Timezones Timezone data dtype label Jun 10, 2018

pep8

c8dabe1

jbrockmendel reviewed Jun 10, 2018

View reviewed changes

Use pydatetime_checkexact

795cc39

jbrockmendel reviewed Jun 11, 2018

View reviewed changes

mroeschke added 2 commits June 11, 2018 20:28

Merge remote-tracking branch 'upstream/master' into correct_normalize

f171f9f

use astimezone instead of normalize

7e183a8

mroeschke commented Jun 12, 2018

View reviewed changes

jreback added this to the 0.23.2 milestone Jun 12, 2018

jreback added Bug Indexing Related to indexing on series/frames, not to indexes themselves labels Jun 12, 2018

jreback requested changes Jun 12, 2018

View reviewed changes

mroeschke added 2 commits June 12, 2018 18:57

Merge remote-tracking branch 'upstream/master' into correct_normalize

3dd00d7

move whatsnew and simplify logic

ae9dc7b

jreback approved these changes Jun 13, 2018

View reviewed changes

jreback added the Needs Backport label Jun 13, 2018

jreback merged commit bc4ccd7 into pandas-dev:master Jun 13, 2018

mroeschke deleted the correct_normalize branch June 13, 2018 15:33

mroeschke mentioned this pull request Jun 15, 2018

TST: Add unit tests for older timezone issues #21491

Merged

7 tasks

david-liu-brattle-1 pushed a commit to david-liu-brattle-1/pandas that referenced this pull request Jun 18, 2018

BUG: Construct Timestamp with tz correctly near DST border (pandas-de…

9e48083

…v#21407)

jorisvandenbossche removed the Needs Backport label Jun 29, 2018

jorisvandenbossche pushed a commit that referenced this pull request Jun 29, 2018

BUG: Construct Timestamp with tz correctly near DST border (#21407)

d537ccb

(cherry picked from commit bc4ccd7)

jorisvandenbossche pushed a commit that referenced this pull request Jul 2, 2018

BUG: Construct Timestamp with tz correctly near DST border (#21407)

d4c48aa

(cherry picked from commit bc4ccd7)

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

BUG: Construct Timestamp with tz correctly near DST border (pandas-de…

dc1d52a

…v#21407)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Construct Timestamp with tz correctly near DST border #21407

BUG: Construct Timestamp with tz correctly near DST border #21407

mroeschke commented Jun 10, 2018 •

edited by jreback

Loading

pep8speaks commented Jun 10, 2018 •

edited

Loading

codecov bot commented Jun 10, 2018 •

edited

Loading

jbrockmendel Jun 10, 2018

mroeschke Jun 11, 2018

jbrockmendel Jun 11, 2018

mroeschke Jun 12, 2018

jreback Jun 12, 2018

jreback Jun 12, 2018

jreback Jun 12, 2018

jreback Jun 12, 2018

jreback commented Jun 13, 2018

BUG: Construct Timestamp with tz correctly near DST border #21407

BUG: Construct Timestamp with tz correctly near DST border #21407

Conversation

mroeschke commented Jun 10, 2018 • edited by jreback Loading

pep8speaks commented Jun 10, 2018 • edited Loading

Comment last updated on June 13, 2018 at 03:17 Hours UTC

codecov bot commented Jun 10, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 13, 2018

mroeschke commented Jun 10, 2018 •

edited by jreback

Loading

pep8speaks commented Jun 10, 2018 •

edited

Loading

codecov bot commented Jun 10, 2018 •

edited

Loading